Sieve Estimators : Consistency and Rates of Convergence

The first term is the estimation error and measures the performance of the discrimination rule with respect to the best hypothesis in H. In previous lectures, we studied performance guarantees for this quantity when ĥn is ERM. Now, we also consider the approximation error, which captures how well a class of hypotheses {Hk} approximates the Bayes decision boundary. For example, consider the histogram classifiers in Figure 1 and Figure 2, respectively. As the grid becomes more fine-grained, the class of hypotheses approximates the Bayes decision boundary with increasing accuracy. The examination of the approximation error will lead to the design of sieve estimators that perform ERM over Hk where k = k(n) grows with n at an appropriate rate such that both the approximation error and the estimation error converge to 0. Note that while the estimation error is random (because it depends on the sample), the approximation error is not random.